Week 15 & before : Ideation & Background #

Video feedback loop #

Currently most of the implementations of the video feedback loop art is done with analog cameras and signals.

The idea is to explore the feasibility of achieving this visual effects by with digital cameras, and therefore digital signal outputs.

Week 16 Feasibility check #

This weeks I mainly focused on the possibility of inputting/outputting analog signals from arduino, as the feasibility verification for the whole project as I think that’s the most difficult part.

I found a library which is written 10 years ago that can generate analog signals to tv called Arduino TV Out.

Github Lib - Arduino TV out

I looked into the examples for this board, seems not very hard but the output signal is also quite limited.

Examples:

• TV-Out-with-Arduino
• Arduino TVout - How to print an image on an TV

I did not found a library for processing analog signal, which is kind of a bummer. Maybe I should completely get rid of the analog signal idea, which will make the project more simple and viable, but also kind of losing the spirit. I’m still thinking about that.

For the feedback loop pattern, I gained inspiration from Floating Points’ mv and oscilloscopes arts. In the next stage I will look into the mechanical structures that I need for generating this type of movements, and also trying to build a minimum viable product of the feedback loop system (with digital signals).

That should contains:

• A cam
• A computer with both video input and output
• Touchdesigner which creates and controls the feedback visuals
• A screen/projector for display

To do list: #

• D/A A/D converter Samuli Kärki’s final project
• Dirty analog video mixer
• Movable Lighting structures in between
• Real-time video processing (probably Touchdesign)

Week 17 & 18 #

Progresses:

• Built the video feedback loop with digital camera and projector
• Tested with regular light source(Neopixel)
• Analyzed the issues currently have and potentially in the future
• Designed and made a new light source board
• Tested it with small scale feedback loop (monitor display and camera)
• Learned how to make feedback and a bunch of other visual effects in touchdesigner

Feedback loop setup #

Given it’s crucial role in the project and my limited knowledge in that area, testing the feedback loop obviously should be prioritized with the utmost importance. The video feedback loop I built for this test contains of three parts:

• A Panasonic Lumix S5 camera
• A projector (the one in Fablab)
• A computer used for process the camera signal and output it onto the projector

Tests #

I conducted test with three “sources” that should create the feedback loop effects.

• Shadow created by human movement
• Neopixel light source
• Led ring light from the lamp

Unfortunately I forgot to record the testing session, but there are some information learned during the process.

Good news:

• You can achieve the video feedback effects with digital devices
• Adding glass or lens in between the camera and the screen could creates some interesting effects.

Bad news:

• In order to achieve strong feedback effects in a regular environment, contrast would need to be tuned up, therefore, a really bright light source would be needed.
• Visual effects are hard to control. The self evolving effects might now be achievable, but simpler effects such as echoing are showing.
• With projectors and the camera placed in the same direction, the light source would cast a shadow on the projector screen which ruins the effect.

New light source #

To address the light sources issues, I made another board which is much brighter than the old one, which is only one pixel. This is achieved by utilizing the brightest LED we have.

According to the datasheet I designed the pcb:

Then I fabricated and tested the board.

It’s bright but the shadow issue still exists. I also tried with the monitor display, but still the pcb is blocking the center part which is crucial for creating the effects.

Next week’s goal:

• explore on the possibilities of making transparent pcb
• design the mechanism for the light source (for the wildcard week also)
• try more feedback set ups, decide which scale would be used for the final one

Week 19 #

Progress:

• Video visual effects
• Transparent circuit board (failed ATM)
• ESP32-CAM
• Mechanical design (in progress)

Visual effects #

I spent some time in touchdesigner, messing around with different visual effects.

This added multiple layers of noise, displacement and color shift which created an illusion of analog video signal for the final demonstration.

I also tested the feedback effect that I was going to use, which is created with altering the rotation angle and focal length.

Transparent PCB #

As a continuation of last weeks’ light source, I tired to make the transparent pcb with acrylic, caption, and copper tape. I tried milling and it was fine.

The issue was when I tried to solder on that, the glue melts and the whole copper pad as well as the track came off. Probably a separate protective mask layer was needed according to Arthur’s final project.

ESP32-CAM #

As the test before was implemented with a webcam, which disrupted my preconception that a DSLR camera is necessary for this project.

So I turned to search for camera modules with higher integration and smaller size. The answer is the ESP32-CAM.

I tested with the webserver cam program. The camera module used was CAMERA_MODEL_AI_THINKER.

#include "esp_camera.h"
#include <WiFi.h>

//
// WARNING!!! PSRAM IC required for UXGA resolution and high JPEG quality
//            Ensure ESP32 Wrover Module or other board with PSRAM is selected
//            Partial images will be transmitted if image exceeds buffer size
//
//            You must select partition scheme from the board menu that has at least 3MB APP space.
//            Face Recognition is DISABLED for ESP32 and ESP32-S2, because it takes up from 15 
//            seconds to process single frame. Face Detection is ENABLED if PSRAM is enabled as well

// ===================
// Select camera model
// ===================
//#define CAMERA_MODEL_WROVER_KIT // Has PSRAM
//#define CAMERA_MODEL_ESP_EYE // Has PSRAM
//#define CAMERA_MODEL_ESP32S3_EYE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_PSRAM // Has PSRAM
//#define CAMERA_MODEL_M5STACK_V2_PSRAM // M5Camera version B Has PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_ESP32CAM // No PSRAM
//#define CAMERA_MODEL_M5STACK_UNITCAM // No PSRAM
#define CAMERA_MODEL_AI_THINKER // Has PSRAM
//#define CAMERA_MODEL_TTGO_T_JOURNAL // No PSRAM
//#define CAMERA_MODEL_XIAO_ESP32S3 // Has PSRAM
// ** Espressif Internal Boards **
//#define CAMERA_MODEL_ESP32_CAM_BOARD
//#define CAMERA_MODEL_ESP32S2_CAM_BOARD
//#define CAMERA_MODEL_ESP32S3_CAM_LCD

#include "camera_pins.h"

// ===========================
// Enter your WiFi credentials
// ===========================
const char* ssid = "Fablab";
const char* password = "Fabricationlab1";

void startCameraServer();
void setupLedFlash(int pin);

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println();

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sccb_sda = SIOD_GPIO_NUM;
  config.pin_sccb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.frame_size = FRAMESIZE_UXGA;
  config.pixel_format = PIXFORMAT_JPEG; // for streaming
  //config.pixel_format = PIXFORMAT_RGB565; // for face detection/recognition
  config.grab_mode = CAMERA_GRAB_WHEN_EMPTY;
  config.fb_location = CAMERA_FB_IN_PSRAM;
  config.jpeg_quality = 12;
  config.fb_count = 1;
  
  // if PSRAM IC present, init with UXGA resolution and higher JPEG quality
  //                      for larger pre-allocated frame buffer.
  if(config.pixel_format == PIXFORMAT_JPEG){
    if(psramFound()){
      config.jpeg_quality = 10;
      config.fb_count = 2;
      config.grab_mode = CAMERA_GRAB_LATEST;
    } else {
      // Limit the frame size when PSRAM is not available
      config.frame_size = FRAMESIZE_SVGA;
      config.fb_location = CAMERA_FB_IN_DRAM;
    }
  } else {
    // Best option for face detection/recognition
    config.frame_size = FRAMESIZE_240X240;
#if CONFIG_IDF_TARGET_ESP32S3
    config.fb_count = 2;
#endif
  }

#if defined(CAMERA_MODEL_ESP_EYE)
  pinMode(13, INPUT_PULLUP);
  pinMode(14, INPUT_PULLUP);
#endif

  // camera init
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

  sensor_t * s = esp_camera_sensor_get();
  // initial sensors are flipped vertically and colors are a bit saturated
  if (s->id.PID == OV3660_PID) {
    s->set_vflip(s, 1); // flip it back
    s->set_brightness(s, 1); // up the brightness just a bit
    s->set_saturation(s, -2); // lower the saturation
  }
  // drop down frame size for higher initial frame rate
  if(config.pixel_format == PIXFORMAT_JPEG){
    s->set_framesize(s, FRAMESIZE_QVGA);
  }

#if defined(CAMERA_MODEL_M5STACK_WIDE) || defined(CAMERA_MODEL_M5STACK_ESP32CAM)
  s->set_vflip(s, 1);
  s->set_hmirror(s, 1);
#endif

#if defined(CAMERA_MODEL_ESP32S3_EYE)
  s->set_vflip(s, 1);
#endif

// Setup LED FLash if LED pin is defined in camera_pins.h
#if defined(LED_GPIO_NUM)
  setupLedFlash(LED_GPIO_NUM);
#endif

  WiFi.begin(ssid, password);
  WiFi.setSleep(false);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  startCameraServer();

  Serial.print("Camera Ready! Use 'http://");
  Serial.print(WiFi.localIP());
  Serial.println("' to connect");
}

void loop() {
  // Do nothing. Everything is done in another task by the web server
  delay(10000);
}

Results are surprisingly good considering the size of the sensor and the board.

Next week’s goal:

• Mechanical design
• Prototype for the cam holder
• Board for the ESP32-cam
• Motor system